Remote object sensing in video

ABSTRACT

A method comprises obtaining an input comprising audio and visual data for display on a first device, receiving data associated with one or more sensory properties of one or more objects in the input, and reconstructing the one or more sensory properties at the first device based on the data received.

FIELD

The present application relates generally to remote object sensing and,more particularly, to techniques for replicating the sense of touchremotely.

BACKGROUND

As technology advances, more and more can be done in the virtualenvironment. For example, technological advances now enable new means ofcommunication, including videotelephony and voice over IP. Communicationbetween family members via remote means using applications such as SKYPEor FACETIME, allow for video and audio transmissions between users inreal-time.

Furthermore, content available for media streaming continually expandsin the virtual environment. Users can now easily send or receive videosand watch movies and shows on a user device. Communication via remotemeans also enable non-traditional forms of education such as distancelearning. Students can now watch live or pre-recorded lectures as wellas explore different subject areas using the vast media contentavailable on the Internet. For example, a child can watch videos oflions in Africa without having to travel to Africa or even to a localzoo; a student in London can participate in a live lecture that istaking place in a New York City classroom.

SUMMARY

Embodiments of the invention provide techniques for recreating the senseof touch remotely using data received at a device.

For example, in one embodiment of the invention, a method comprisessteps of obtaining an input comprising audio and visual data for displayon a first device, receiving data associated with one or more sensoryproperties of one or more objects in the input, and reconstructing theone or more sensory properties at the first device based on the datareceived.

In additional embodiments of the invention, a method comprises obtainingan input comprising visual and audio data for display on a device,detecting user selection of one or more objects in the visual input,identifying the one or more selected objects and surroundings of the oneor more objects, obtaining data associated with one or more sensoryproperties of the identified one or more objects and the surroundings,and reconstructing the one or more sensory properties associated withthe identified one or more objects and the surroundings based on thedata received.

In further embodiments of the invention, a method comprises obtaining aninput comprising visual and audio data for display on a first device,detecting sensory input from a user of the first device, capturing dataassociated with the sensory input from the user of the first device, andtransmitting the captured data to a second device for reconstruction ofone or more sensory properties associated with the user of the firstdevice.

In further embodiments, an apparatus comprises a memory and a processoroperatively coupled to the memory. The processor is configured to obtainan input comprising audio and visual data for display on a first device,receive data associated with one or more sensory properties of one ormore objects in the input, and reconstruct the one or more sensoryproperties at the first device based on the data received.

These and other objects, features, and advantages of the presentinvention will become apparent from the following detailed descriptionof illustrative embodiments thereof, which is to be read in connectionwith the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an overview of a remote sensing methodology according toan embodiment of the invention.

FIG. 2 depicts a remote sensory module for the remote sensingmethodology of FIG. 1 according to an embodiment of the invention.

FIG. 3 depicts a coordinated plane used in the remote sensing module ofFIG. 2, according to an embodiment of the invention.

FIG. 4 depicts a controller used in the remote sensing module of FIG. 2,according to an embodiment of the invention.

FIG. 5 depicts an additional embodiment of the remote sensory module ofthe remote sensing methodology of FIG. 1, according to an embodiment ofthe invention.

FIG. 6 depicts a sensory estimator used in the remote sensory module ofFIG. 5, according to an embodiment of the invention.

FIG. 7 depicts a further embodiment of the remote sensory module of theremote sensing methodology of FIG. 1.

FIG. 8A depicts an example of a front portion of a device with a griddedinterface for implementing the remote sensing methodology according toan embodiment of the invention.

FIG. 8B depicts a back portion of the device of FIG. 8A.

FIG. 9 depicts an exemplary application of the remote sensingmethodology according to embodiments of the invention.

FIG. 10 depicts a computer system in accordance with which one or morecomponents/steps of techniques of the invention may be implementedaccording to an embodiment of the invention.

DETAILED DESCRIPTION

As visual and/or auditory information is used in conventional computertechnology, users demand more and more specific and realisticinformation. However, an essential element is still missing fromexisting means of virtual communication and content streaming—the senseof touch. As a result, there is a growing need to replicate the sense oftouch remotely. Replicating the sense of touch would be useful in avariety of contexts. For example, if a parent wants to touch their childfrom work to see if the child has a fever or if there is a change in thechild's body temperature. As another example, if a doctor wants to checka patient's heartbeat or other vitals remotely. As a further example, itwould be desirable to interact with a visual input by “feeling around”the imagery displayed on the screen of a user device, and remotelyfeeling the objects and the surroundings displayed on the device (e.g.,feeling the fabric of each piece of clothing while shopping online).

Illustrative embodiments of the invention provide for replicating asense of touch by augmenting existing audio and/or visual content with asensory dimension in the form of temperature, texture and movement. Forexample, embodiments of the invention augment traditional videotelephonyand video recordings by conveying the sense of warmth or pressure from ahug or a hand shake, the sensation of snow, the feeling of a patient'spulse and body temperature, etc.

Illustratively, embodiments of the invention replicate a sense of touchremotely by recreating roughness and/or temperature sensation. It isbelieved that a combination of the three remote patterns—sense oftexture, temperature and movement, would allow for a more accuraterepresentation of the sense of touch.

FIG. 1 shows an overview of the remote sensing methodology according toillustrative embodiments of the invention. Remote sensing methodology100 starts at block 102 in which data is obtain as input for display ona user interface such as a screen on a device. For example, the devicemay be a handheld device or any suitable computing or electronic device.Non-limiting examples of input can include audio and visual input suchas a video, a still image, an audio clip or a live feed. Then at block104, a remote sensory module performs operations on the data received asinput. Details of the remote sensory module 104 are further described inthe context of FIGS. 2 through 7. Finally, at block 106, output from theremote sensory module is reproduced for the user at the user interface.

FIG. 2 shows an illustrative embodiment of the remote sensory module 104of FIG. 1. At block 202, visual and/or audio input is received. Thevisual and/or audio input can be pre-recorded content or live streamingcontent. For example, the input may be a recording of a patient for thepast twenty-four hours or any given duration, or a live stream of thepatient in his room. As another example, the input may be a live videoconference between two or more users. As a further example, the inputmay be captured by a visual or audio recording component on a user'sdevice (e.g., the camera on a mobile device). At block 204, datarelating to one or more sensory properties for one or more objects inthe visual and/or audio input is received. As used herein, a sensoryproperty refers to a temperature, a texture or movement associated withthe object. As used herein, an object in the visual and/or audio inputrefers to a person, an animal, an inanimate object (e.g., a fireplace, acar) or the background (e.g., snow, wind, book shelves). Sensory dataassociated with the one or more objects of the input may be receivedfrom one or more sensors on the user's device (e.g., a camera on amobile phone, a tablet, a computer). Alternatively, sensory dataassociated with the one or more objects of the input may be receivedfrom a second device, such as sensory data captured by one or moresensors on the second device. As used herein, sensors may refer toinfrared sensors, movement detectors, tactile sensors, etc. At block206, the data associated with one or more sensory properties of the oneor more objects in the input are processed by the coordinated plane andsent to the controller at block 208. The coordinated plane will bedescribed in further detail in the context of FIG. 3 below. At block208, the controller coordinates and controls reconstruction andrecreation of the one or more sensory properties associated with the oneor more objects in the input. The reconstructed sensory properties canthen be experienced by the user at a user device. Details of controller208 will be further described in the context of FIG. 4 below.

An illustrative embodiment of the coordinated plane 206 of FIG. 2 isshown in FIG. 3. Coordinated plane 300 includes block 302, at which datais received. The data received corresponds to data associated with theone or more sensory properties of the one or more objects in the inputpreviously described in the context of FIG. 2. The data received may bein the form of temperature data, electrical signals, vibration durationand frequency, etc. At block 304, the sensory data is organized and/orinterpreted. Then at block 306, the data is translated into metrics suchthat the one or more sensory properties associated with the one or moreobjects in the input may be replicated. The translated metrics are thensent to the controller at block 308 for reconstruction. For example, thebody temperature of a person's hand may be received at block 302,translated as the amount of heat necessary to bring the heating elementsto the given temperature at blocks 304 and 306 and this translatedmetric is then sent to controller at block 308. Additionally,coordinated plane 300 may include a calibration algorithm, such thatcalibration can be performed depending on the input. For example, if theinput is that of a dessert scene, the algorithm would automaticallycalibrate the coordinated plane 300 to a hot climate module.

FIG. 4 shows an illustrative embodiment of the controller 208 of FIG. 2.Controller 400 coordinates and controls reconstruction of the one ormore sensory properties associated with the one or more objects in theinput. Controller 400 includes a temperature simulator 402, a texturesimulator 404 and a movement simulator 406. Temperature simulator 402may comprise elements capable of heating and cooling, so that variationsin temperature can be produced by means such as electromagneticinduction, laser and air flow. Texture simulator 404 may compriseelements capable of producing vibrations at various frequencies ormaterials that are capable of morphing (such as phase change materials)giving rise to a feeling of roughness. Movement simulator 406 maycomprise elements capable of generating force and pressure to emulatemotion or micro-accelerators and micro-sensors, Controller 400 may beimplemented as software capable of controlling and activatingtemperature simulator 402, texture simulator 404 and movement simulator406 to reconstruct the sensory properties associated with the one ormore objects in the input. The temperature simulator 402, texturesimulator 404 and movement simulator 406 may be located on the enclosureof a user device, such as the back of the device, the screen of thedevice or a case suitable for enclosing a back and/or front of thedevice.

As an illustrative example of the remote sensing methodology asdescribed in FIGS. 1 to 4, a doctor is interested in monitoring apatient for a specific duration of time. Corresponding to block 102 ofFIG. 1 and 202 of FIG. 2, a first device obtains visual and/or audioinput relating to the patient for display on the first device or on asecond device. Sensors on the first device, such as an infraredthermometer, a movement sensor, and a tactile sensor, can be used tomonitor and record the patient's vitals or other parameters of interest.Data from the sensors can be captured and stored in a database foranalysis at a later time, or it can be analyzed simultaneously at thesame device, or sent to a second device for use. Sensory data to becaptured and recorded can be specified by the doctor prior to monitoringthe patient. Then at block 104 of FIG. and block 204 of FIG. 2, thesensory data relating to the patient is received for processing. Atblock 206 of FIG. 2, also corresponding to the coordinated plane 300 ofFIG. 3, the data is processed and sent to a controller at block 208. Atblock 208 and corresponding controller 400 of FIG. 4, the controllerreconstructs and replicates the one or more sensory properties from thedata received by activating and controlling the texture simulator 402,texture simulator 404 and/or movement simulator 406. Corresponding toblock 106, the reconstructed sensory properties can be experienced bythe doctor at a user device. For example, if the doctor was interestedin monitoring the patient's temperature and heart rate, the patient'stemperature and heart rate would be reconstructed at the device on whichthe doctor is using and the doctor would feel the body heat and thepulse of the patient at a given point in time.

As another illustrative example of the remote sensing methodology asdescribed in FIGS. 1 to 4, a child is be at a zoo and points the cameraof a mobile device at a lion (corresponding to obtaining visual and/orauditory input at blocks 102 and 202). Infrared sensors on the mobiledevice can establish the lion's temperature, especially relative to thebackground; tactile sensors on the mobile device can capture texture soas to communicate what the lion feels like; movement detectors on themobile device can capture the lion's periodic movement (e.g., as inpulsing). This data is received at blocks 104 and 204. The coordinatedplane of block 206 and FIG. 3 then interprets and translates thecaptured sensory data. The translated sensory data is then sent to thecontroller of block 208 (correspondingly controller 400 of FIG. 4) forreconstruction. The reconstructed sensory properties are thenexperienced as the output at block 106. Therefore, as the child looks atthe lion being recorded through the camera and displayed on the mobiledevice, through a sensory replication interface on the mobile device,the child can also feel the lion's heat, feel its texture and itsmovement on an interface at his mobile device, thus giving a veryrealistic sense of the lion. The interface will be described in furtherdetails in the context of FIGS. 8A and 8B below.

FIG. 5 shows an alternative embodiment of the remote sensory module 104of FIG. 1. At block 502, visual and/or audio input is obtained fordisplay on a user device. The visual and/or audio input can bepre-recorded content or live streaming content. For example, the inputmay be a pre-recorded show or live streaming content from the camera ofa device. At block 504, a user may select a specific object and/orsensory property for reconstruction through the user interface of theuser device. At block 506, the user may optionally choose to segment andpartition the input by duration (temporal variation) or by object/region(spatial variation). At block 508, a sensory estimator generates orobtains data based on the user selection. The data is then sent to block510, where the coordinated plane (as disclosed in FIG. 3) interprets thedata and translates it into metrics for reconstruction by a controller(as disclosed in FIG. 4) at block 512.

FIG. 6 shows an illustrative embodiment of the sensory estimator 508 ofFIG. 5. Sensory estimator 600 includes temperature estimator 604,texture estimator 606 and movement estimator 608. Sensory estimator 600interacts with database 602 to obtain sensory data for the one or moreselected objects or regions. Database 602 was created at an earlierpoint in time with objects indexed with sensory information, much like alibrary having numbers and vibrations that have been translated into alexicon or dictionary of temperatures, textures and movement thatmatches the physical experience of the associated object. Database 602continually updates and expands as new information is acquired.Information in database 602 may come from the data captured by one ormore devices with access to the database 602, entered by a user (e.g.,the temperature of boiling water is approximately 100° C.), as well asacquired from online resources such as the World Wide Web. Sensoryestimator 600 interacts with database 602 to retrieve data related toone or more sensory properties of an object when desired or needed. Forexample, when a user is interested in an average temperature of aspecific object or region in the input, sensory estimator 600 can obtaintemperature information of the object or region for a given duration andcalculate an average through temperature estimator 604. As anotherexample, a user may stream a pre-recorded show in which no sensory datais transmitted and associated with the input. When the user selects anobject or region of interest from the input, such as an article ofclothing or an animal, sensory estimator 600 interacts with database 602to identify the object or region of interest (e.g., via an imagerecognition module within the database or by querying the user foridentification of the object). Once the object or region of interest isidentified, database 602 sends sensory information, such as temperature,texture and movement, associated with the identified object/region tothe respective estimators of sensory estimator 600. If the object is notfound in database 602, a search outside the database may be performed toidentify the object. If sensory information for the object is not foundin database 602, then database 602 may identify related and similarobjects (e.g., a fire pit is similar to a fireplace) and send sensorydata associated with the related objects to sensory estimator 600.

As an example of the remote sensory module as described in FIGS. 5 and6, in the previous example in which a doctor is interested in monitoringa patient, once visual and/or audio input is obtained for display on theuser's device at block 502, the doctor can select the parameters ofinterest to reconstruct. This selection is detected at block 504. Thedoctor may choose to monitor how a patient's temperature evolve overtime, and thus select segmentation of the input by durationcorresponding to block 506. Alternatively, the doctor may choose topartition the input by object such that in an entire room of patients,the doctor can select a specific patient or a specific section of thepatient room to monitor. At block 508 and corresponding sensoryestimator of FIG. 6, the sensory estimator may obtain informationassociated with the one or more sensory properties of the one or moreobjects in the input. In the instance where the doctor wants to monitorhow the patient's temperature evolve over time (e.g., the temperaturechange per hour for the past three hours to detect change in fever inresponse to a medication), sensory estimator 600 can obtain temperatureinformation for the patient over the given duration from database 602.Temperature estimator 604 can then calculate an average temperature forthe patient for each hour of the duration of interest. Sensory estimator600 then sends this average temperature data to coordinated plane 510.Coordinated plane 510 interprets and translates the temperature data asdescribed in the context of FIG. 3 above and sends the metrics tocontroller 512. Controller 512 reconstructs the average temperature foreach of the three hours as output for the doctor to experience at hisdevice. For example, if the average temperature for the three hours were101.6° F., 100.2° F. and 98.8° F., controller 512 may control theheating element(s) to reproduce the heat feel of these temperatures forthe doctor to experience. The doctor can choose to experience theaverage temperature for each of the three hours for a length of time(e.g., five seconds per average temperature) consecutively orindependently. Additionally, the controller 512 can also use the tactileand/or movement simulators to supplement the temperature feel of thepatient with the pulse rate of the patient for the same duration ofinterest.

FIG. 7 shows an additional embodiment of the remote sensory module 104of FIG. 1. At block 702, visual and/or audio input is obtained fordisplay on a device. The visual and/or audio input is preferably livestreaming content, for example, a video chat between two or more people.At block 704, a determination is made whether sensory input is detected.For example, a mother and her child are chatting remotely via a videochat application such as SKYPE or FACETIME, the mother wants to give thechild a reassuring gesture such as a pat or hand squeeze. The mother canmake contact with the sensors on her device as if she is actuallysqueezing the child's hand or patting the child on the hand. If nosensory input is detected, then at block 706 the module returns to block702. If sensory input is detected at block 708, the module proceeds toblock 710 in which sensory data associated with the gestures arecaptured (e.g., the temperature and texture of the hand, the movementand pressure associated with the gesture). The captured data associatedwith the various sensory properties of the gesture is then transmittedto a device of an intended recipient at block 712 for reconstruction atthe recipient's device, here, transmitted to the child's device forreconstruction. Reconstruction can be performed at the recipient'sdevice using a remote sensory module as according to embodiments of theinvention, for example, as described in the context of FIGS. 2 to 6above.

FIGS. 8A and 8B show an illustrative embodiment of a device capable ofimplementing the remote sensing methodology 100 as shown in FIG. 1. Thedevice may be any handheld or portable device (e.g., smartphone, tablet,thermometer) or any other suitable device (e.g., personal computer,dashboard of an automobile). Sensors for capturing sensory data andsimulators for reproducing one or more sensory properties may be placedanywhere along the enclosure of the device, for example, on the screenand/or back of a tablet or smart phones. The sensors and simulators maybe part of a gridded interface spanning a desired portion of theenclosure of the device.

FIG. 8A shows the front of a device 802, with screen 810. A griddedinterface 808 spans at least a portion of the screen 810 of device 802.The gridded interface 808 may be embedded into the screen 810 of device802 or embedded into an enclosure, such as a case or cover for device802 that covers the screen 810. The gridded interface 808 may comprise aminiscule grid with a network of micro pipes for delivery of heating andcooling (e.g., using hot air and cold air stored within reservoirs inthe device or provided as part of the gridded interface), illustrated asdashed lines 804-1 . . . 804-M. By controlling this network of micropipes, the controller can create an appropriate mixture of hot air andcold air to provide the appropriate temperature sensation. The number ofmicro pipes, M, may vary accordingly, for example, depending on the sizeof the device or area of desired coverage on the device. Preferably, thenetwork of micro pipes span the entire surface of a device. The griddedinterface 808 may also comprise an array of tactile and movement sensorsand simulators, illustrated as ovals 806-1 . . . 806-N. (e.g., micromotors for generating vibrations at various frequencies to simulatedifferent textures). The number, size and arrangement of the tactileand/or movement sensors and simulators may vary accordingly, forexample, depending on the size of the device. The gridded interface 808may be activated electronically and locally by the controller module.Accordingly, through an option in the user interface, the griddedinterface 808 may be activated and continuously capturing, transmittingand/or receiving sensory data, or the gridded interface 808 may be insleep mode such that sensory data is only captured, transmitted and/orreceived when re-activated.

Similarly, FIG. 8B shows the gridded interface 808 implemented on atleast a portion of the back of device 802. It is to be noted that thegridded interface described herein may be implemented in a front portionof device 802, a back portion of device 802 or in the front and backportions of device 802.

Alternatively, the gridded interface 808 may be implemented apart from adevice as a stand-alone apparatus through which a device may be pluggedin when sensory replication is desired or needed.

FIG. 9 shows an exemplary application of embodiments of the invention. Avisual and/or audio input is displayed on screen 902. Screen 902 may bea screen on any suitable device, such as a tablet or a mobile phone. Asillustratively shown, displayed on screen 902 is an outdoor scene.Sensory properties, such as temperature, texture and/or movementassociated with objects in the scene, such as the sun 904, snow 906 andfire 908, may be experienced by a user of the device.

In the instance where the scene is a live feed captured by a cameracomponent on a device, the sensors on the device may capture sensorydata associated with the objects in the scene. For example, thetemperature and texture of the snow 906, the temperature and movement ofthe fire 908, along with the temperature of the sun 904 may be capturedby the temperature sensor, tactile sensor and motion sensor on thedevice. These sensory data can then be sent to coordinated plane 910(details of which are disclosed in the context of FIG. 3 above), whichinterprets and translates the received sensory data into metrics andsends the translated metrics to controller 912 (details of which aredisclosed in the context of FIG. 4 above). Controller 912 then activatesand coordinates the temperature, texture and movement simulators on thedevice to reconstruct the received sensory data into one or more sensoryproperty associated with the sun 904, snow 906 and fire 908. The sensoryproperties can be experienced by the user on the device through thegridded interface 808 as described in FIGS. 8A and 8B.

A user may also chose to experience only one sensory property associatedwith the objects in the scene. For example, a user may choose toexperience only the temperature of the objects in the scene. In thatinstance, infrared sensors on the device can be used to sense thetemperature of the sun 904, snow 906 and fire 908. It should be notedthat while the true temperature of the sun 904 or of the fire 908 maynot be remotely sensed, the perceived temperature can readily becaptured by the temperature sensors on the device. Accordingly, anexemplary output produced by the controller 912 may be a relativetemperature map of the scene such that portions of the image is muchwarmer than the surroundings. Here, the sun 904 would be perceived aswarm relative to the hot fire 908, and the snow 906 would be coldrelative to the sun 904 and fire 908. A user would be able to touch thescreen or back of the device, depending on where the gridded interfaceis located, to feel the temperature distribution of the scene displayedon screen 902.

In the instance where the scene is an input from pre-recorded content inwhich sensory data is not transmitted along with the input (e.g., amovie), the remote sensing methodology 100 would replicate the sense oftouch using a remote sensory module 500 as described in FIG. 5. Sensorydata associated with the one or more objects and the surroundings in thescene displayed on screen 902 would be obtained from a database such asdatabase 602 of FIG. 6. Furthermore, the user may choose to segment theinput by duration or object or region. If the user selects an object forwhich the database may not have sensory information, such as the cloudin the scene, a sensory estimator, such as sensory estimator 600 of FIG.6, may estimate the texture, temperature and/or movement of the cloud.

In an alternative embodiment, screen 902 may be a window through whichthe scene is being viewed. The window 902 may contain a griddedinterface 808, as described in FIGS. 8A and 8B, embedded into the windowpane. A user may, for example, perceive the temperature feel of thescene outside by touching the various regions on the window pane, whichis reproduced by the simulators of the gridded interface. As such, auser may “feel” the temperature outside without having to leave thehouse or check the weather reports.

It should be noted that where user selection is an option, such as userselection of the object of interest or user selection of segmentation byduration, the user selection may be performed using conventional methodssuch as employing a stylus, a mouse or a finger to highlight the objector choice in the user interface.

Embodiments of the present invention may be a system, a method, and/or acomputer program product. The computer program product may include acomputer readable storage medium (or media) having computer readableprogram instructions thereon for causing a processor to carry outaspects of the present invention.

Accordingly, the architecture shown in FIG. 10 may be used to implementthe various components/steps shown and described above in the context ofFIGS. 1-9.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

One or more embodiments can make use of software running on ageneral-purpose computer or workstation. With reference to FIG. 10, in acomputing node 1010 there is a computer system/server 1012, which isoperational with numerous other general purpose or special purposecomputing system environments or configurations. Examples of well-knowncomputing systems, environments, and/or configurations that may besuitable for use with computer system/server 1012 include, but are notlimited to, personal computer systems, server computer systems, thinclients, thick clients, handheld or laptop devices, multiprocessorsystems, microprocessor-based systems, set top boxes, programmableconsumer electronics, network PCs, minicomputer systems, mainframecomputer systems, and distributed cloud computing environments thatinclude any of the above systems or devices, and the like.

Computer system/server 1012 may be described in the general context ofcomputer system executable instructions, such as program modules, beingexecuted by a computer system. Generally, program modules may includeroutines, programs, objects, components, logic, data structures, and soon that perform particular tasks or implement particular abstract datatypes. Computer system/server 1012 may be practiced in distributed cloudcomputing environments where tasks are performed by remote processingdevices that are linked through a communications network. In adistributed cloud computing environment, program modules may be locatedin both local and remote computer system storage media including memorystorage devices.

As shown in FIG. 10, computer system/server 1012 in computing node 1010is shown in the form of a general-purpose computing device. Thecomponents of computer system/server 1012 may include, but are notlimited to, one or more processors or processing units 1016, a systemmemory 1028, and a bus 1018 that couples various system componentsincluding system memory 1028 to processor 1016.

The bus 1018 represents one or more of any of several types of busstructures, including a memory bus or memory controller, a peripheralbus, an accelerated graphics port, and a processor or local bus usingany of a variety of bus architectures. By way of example, and notlimitation, such architectures include Industry Standard Architecture(ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA)bus, Video Electronics Standards Association (VESA) local bus, andPeripheral Component Interconnects (PCI) bus.

The computer system/server 1012 typically includes a variety of computersystem readable media. Such media may be any available media that isaccessible by computer system/server 1012, and it includes both volatileand non-volatile media, removable and non-removable media.

The system memory 1028 can include computer system readable media in theform of volatile memory, such as random access memory (RAM) 1030 and/orcache memory 1032. The computer system/server 1012 may further includeother removable/non-removable, volatile/nonvolatile computer systemstorage media. By way of example only, storage system 1034 can beprovided for reading from and writing to a non-removable, non-volatilemagnetic media (not shown and typically called a “hard drive”). Althoughnot shown, a magnetic disk drive for reading from and writing to aremovable, non-volatile magnetic disk (e.g., a “floppy disk”), and anoptical disk drive for reading from or writing to a removable,non-volatile optical disk such as a CD-ROM, DVD-ROM or other opticalmedia can be provided. In such instances, each can be connected to thebus 1018 by one or more data media interfaces. As depicted and describedherein, the memory 1028 may include at least one program product havinga set (e.g., at least one) of program modules that are configured tocarry out the functions of embodiments of the invention. Aprogram/utility 1040, having a set (at least one) of program modules1042, may be stored in memory 1028 by way of example, and notlimitation, as well as an operating system, one or more applicationprograms, other program modules, and program data. Each of the operatingsystem, one or more application programs, other program modules, andprogram data or some combination thereof, may include an implementationof a networking environment. Program modules 1042 generally carry outthe functions and/or methodologies of embodiments of the invention asdescribed herein.

Computer system/server 1012 may also communicate with one or moreexternal devices 1014 such as a keyboard, a pointing device, a display1024, etc., one or more devices that enable a user to interact withcomputer system/server 1012, and/or any devices (e.g., network card,modem, etc.) that enable computer system/server 1012 to communicate withone or more other computing devices. Such communication can occur viaInput/Output (I/O) interfaces 1022. Still yet, computer system/server1012 can communicate with one or more networks such as a local areanetwork (LAN), a general wide area network (WAN), and/or a publicnetwork (e.g., the Internet) via network adapter 1020. As depicted,network adapter 1020 communicates with the other components of computersystem/server 1012 via bus 1018. It should be understood that althoughnot shown, other hardware and/or software components could be used inconjunction with computer system/server 1012. Examples include, but arenot limited to, microcode, device drivers, redundant processing units,external disk drive arrays, RAID systems, tape drives, and data archivalstorage systems, etc.

Although illustrative embodiments of the present invention have beendescribed herein with reference to the accompanying drawings, it is tobe understood that the invention is not limited to those preciseembodiments, and that various other changes and modifications may bemade by one skilled in the art without departing from the scope orspirit of the invention.

What is claimed is:
 1. A method, comprising: obtaining an inputcomprising audio and visual data for display on a first device;receiving data associated with one or more sensory properties of one ormore user-selected objects in the input; and reconstructing the one ormore sensory properties at the first device based on the data received.2. The method of claim 1, wherein the one or more sensory propertiescomprises at least one of a temperature, a texture and a movementassociated with the one or more.
 3. The method of claim 1, wherein thedata is captured by one or more sensors on the first device, the one ormore sensors comprising at least one of an infrared sensor and a tactilesensor.
 4. The method of claim 1, wherein the data is captured by one ormore sensors on a second device, the one or more sensors comprising atleast one of an infrared sensor and a tactile sensor.
 5. The method ofclaim 1, wherein reconstructing the one or more sensory propertiescomprises translating the sensory data into metrics for reconstruction.6. The method of claim 5, wherein a controller regulates a griddedinterface to produce the one or more sensory properties based on themetrics.
 7. The method of claim 6, wherein the gridded interfacecomprises at least one of an array of micro pipes, a hot air reservoir,a cold air reservoir and a tactile simulator.
 8. The method of claim 7,wherein the gridded interface is located on an enclosure of the firstdevice.
 9. The method of claim 1, wherein the input is one ofpre-recorded or live.
 10. The method of claim 1, further comprisingsegmenting the input based on user selection, wherein the input is atleast one of segmented temporally and segmented spatially.
 11. Themethod of claim 10, wherein the one or more sensory properties arereconstructed based on the user selection for at least one of a givenduration and a given region of the input.
 12. A method, comprising:obtaining an input comprising visual and audio data for display on adevice; detecting user selection of one or more objects in the visualinput; identifying the one or more selected objects and surroundings ofthe one or more objects; obtaining data associated with one or moresensory properties of the identified one or more objects and thesurroundings; and reconstructing the one or more sensory propertiesassociated with the identified one or more objects and the surroundingsbased on the data received.
 13. The method of claim 12, whereinidentifying the one or more selected objects comprises one of an imagerecognition software and a user input.
 14. The method of claim 12,wherein the data is obtained from a database comprising objects indexedwith sensory information.
 15. The method of claim 14, wherein the datais estimated based on historical information stored in the database. 16.The method of claim 12, wherein the sensory properties comprises atleast one of a temperature, a texture and a movement associated with theone or more objects and surroundings.
 17. The method of claim 12,wherein reconstructing the one or more sensory properties comprisestranslating the data into metrics.
 18. The method of claim 17, wherein acontroller regulates a gridded interface to produce the one or moresensory properties based on the metrics.
 19. A method, comprising:obtaining an input comprising visual and audio data for display on afirst device; detecting sensory input from a user of the first devicewith respect to one or more user-selected objects in the input;capturing data associated with the sensory input from the user of thefirst device; and transmitting the captured data to a second device forreconstruction of one or more sensory properties associated with theuser of the first device.
 20. The method of claim 19, wherein detectingsensory input from a user comprises detecting user contact with one ormore sensors on the first device.
 21. The method of claim 19, whereincapturing data associated with the sensory input comprises recording andstoring data generated by the one of more sensors on the first device.22. An apparatus, comprising: a memory; and a processor operativelycoupled to the memory and configured to: obtain an input comprisingaudio and visual data for display on a first device; receive dataassociated with one or more sensory properties of one or moreuser-selected objects in the input; and reconstruct the one or moresensory properties at the first device based on the data received. 23.The apparatus of claim 22, wherein a controller regulates a griddedinterface to produce one or more sensory properties based on one or moremetrics.
 24. The apparatus of claim 23, wherein the gridded interfacecomprises at least one of an array of micro pipes, a hot air reservoir,a cold air reservoir and a tactile simulator.
 25. The apparatus of claim24, wherein the gridded interface is located on an enclosure of thefirst device.